This application is related to U.S. patent applications entitled xe2x80x9cSwitch Adapter Testingxe2x80x9d and xe2x80x9cError Injection Apparatus and Method,xe2x80x9d filed on even date, which are assigned to the assignee of the present patent application and are incorporated herein by reference.
The present invention relates generally to packet-switched computer networks, and specifically to methods and apparatus for testing and diagnosing malfunctions in such networks.
Packet-switched, source-routing computer networks are used in a growing range of applications. Such networks link multiple computer processors, or nodes, via multiple switches. Typically, a packet of data sent from one of the nodes to another passes through a number of different switches. Each switch along the way reads routing information, which is commonly contained in a header of the data packet, and passes the packet on to the next switch along the way, or to the destination node. Typically, there are multiple different paths available through the network over which any given pair of nodes can communicate. An example of this type of network is the well-known Asynchronous Transfer Mode (ATM) network, which is used in communications between separate computers. Such networks are also used in multi-processor computers, such as the RS/6000 Scalable POWERParallel System (SP) series of computers produced by International Business Machines Corporation (Armonk, N.Y.). In the SP computer, as well as in certain other networks, successive packets in a communication stream between the nodes may be sent over different routes.
Because of the complex topology and hardware of packet-switched networks, when a fault occurs in such a network it can be difficult to identify the exact location and nature of the fault. The difficulty is exacerbated by the fact, noted above, that by their nature such networks use multiple different paths between nodes and are fault-tolerant. A network fault will typically appear not as a total breakdown (which would be relatively easy to find), but rather will present more subtle symptoms. For example, there may be a reduction in throughput between some or all of the nodes, or an increase in the number of xe2x80x9cbad packetsxe2x80x9dxe2x80x94data packets whose content is corrupted and must be discardedxe2x80x94at one or more of the nodes.
There are few efficient tools known in the art for diagnosis of such faults. The diagnostic process is time-consuming and heavily reliant on the intuition and experience of a human system administrator (or service engineer) in deciphering and drawing conclusions from the limited information that is available. This information is typically collected in various system files, such as topology files, error logs and trace files, as are known in the art. These files may be recorded at different nodes of the network and must somehow be collated and analyzed by the administrator. Because few network administrators have the know-how to perform this sort of diagnosis, costly service calls are frequently required.
A further problem in diagnosing network faults is non-deterministic failures, which may occur only under certain conditions, and may not arise at all while the diagnostic tests are being performed. Such failures are referred to with terms such as xe2x80x9csporadic,xe2x80x9d xe2x80x9cintermittent,xe2x80x9d xe2x80x9coverheating,xe2x80x9d xe2x80x9clightning,xe2x80x9d xe2x80x9caging,xe2x80x9d or xe2x80x9cstatics,xe2x80x9d which generally mean only that the cause of the problem is unknown. For example, a high-speed switch or adapter may behave normally in light traffic, and break down only under certain particular stress conditions. At times the only way to find such a problem is to systematically bombard each suspect component of the network with packets from different sources, at controlled rates, gradually eliminating components from consideration until the failure is found. Such a process is difficult to automate, and may require that the network be taken off-line for an extended period. The cost of such down-time for prolonged testing and repair can be enormous. There is therefore a need for systematic methods of diagnostic testing, which can be performed while the network is on-line.
There is a similar lack of tools and techniques for systematically testing the response of switch-related network software to hardware fault conditions. Such techniques are needed particularly in software development and testing stages, to ensure that the software responds properly when faults occur. Current methods of testing use specially-designed simulation hardware, such as cables with broken pins, together with debugging clauses that can be activated in the software itself and dedicated debugging fields in associated data structures. The fault situations created by such methods, however, are limited to a small range of scenarios, which are for the most part different from the real hardware faults that occur in actual networks. Similarly, the software used in debugging mode for fault simulation is different from the actual software product that will be used in the field. Moreover, these testing tools are incapable of simulating the type of transient, non-deterministic failures described above. They do not allow errors to be injected and altered on the fly during a simulation.
It is an object of some aspects of the present invention to provide improved methods for fault simulation and diagnostics in packet-switched data networks.
It is a further object of some aspects of the present invention to provide diagnostic methods and apparatus for use in testing a switch in a network while the network is on-line.
Preferred embodiments of the present invention operate in the context of a packet data network, which comprises a plurality of nodes, or processors, mutually coupled by a plurality of switches, such that typically any one of the nodes can communicate with any other one of the nodes, preferably over multiple links. Each of the nodes is coupled to a respective port of one of the switches by a switch adapter, which performs data link functions, as are known in the art, with respect to each data packet sent or received through the network by the node. One of the nodes is a primary node, which manages the configuration of elements of the network, such as the other nodes and switches in the network.
In preferred embodiments of the present invention, the primary node controls testing and diagnosis of elements of the network in real time, while the network is on-line, or at least with minimal interruption of on-line operation, by appropriately setting parameters of the nodes and switches. The testing preferably includes diagnostic testing to locate suspected faults in the switches and switch adapters. Additionally or alternatively, for the purposes of testing, errors are intentionally injected into the network so as to simulate the response of the network elements to faults that may occur.
In some preferred embodiments of the present invention, a switch that is suspected of being faulty is isolated from the remainder of the network for testing, while the remainder of the network continues to operate in a substantially normal fashion. The suspect switch has multiple ports, each of which may be connected either to a node or to a port of another switch. Data packets are passed through the switch via different pairs of the ports, at controllable, preferably high rates, until conditions are created in which the switch fails, or until it is ascertained that the switch is not faulty. By isolating the suspect switch in this manner, it is possible to test the switch quickly and thoroughly, while minimizing interruption to the rest of the network.
Preferably, for every port of the suspect switch, the primary node assigns one of the nodes of the network to be an assistant node. Preferably, the assistant nodes are chosen to be the nodes closest in the network to the respective ports of the suspect switch. The assistant nodes do not need to be connected directly to the respective ports, but may rather be connected through other switches in the network. The assistant nodes are taken off-line in order to isolate the suspect switch. They are then controlled by the primary node to transmit heavy data traffic through the suspect switch, either through specific pairs or combinations of ports, or through all of the ports at once. The primary node receives from the assistant nodes records of the data packets that they have sent and/or received through the suspect switch and analyzes the records for any discrepancies, which are indicative of the fault in the switch.
Preferably, the primary node assigns the links over which the assistant nodes are to communicate with their respective ports on the suspect switch by downloading appropriate entries to a route table of the switch adapter of each of the assistant nodes. Typically, in normal network operation, such a route table includes several different routing links over which the node is to communicate, and the switch adapter sends data packets in alternation over the different links in order to balance the data traffic load among different switches and ports in the network. In preferred embodiments of the present invention, however, the entries downloaded to the route tables associated with the assistant nodes indicate that all of the packets are to be sent over the same links, so as to control and maximize the traffic load on the ports of the suspect switch.
There is therefore provided, in accordance with a preferred embodiment of the present invention, in a computer network system that includes a multiplicity of nodes interconnected by a network of switches, each switch having multiple ports, a method for testing one of the switches, including:
assigning a respective one of the nodes to each of a plurality of the ports of the switch being tested;
configuring the network so that the switch being tested is substantially isolated from data flow from and to the nodes in the network that are not assigned to the ports of the switch;
conveying data at a controlled rate between two or more of the assigned nodes through the respective ports; and
observing behavior of the switch in response to the data being conveyed.
Preferably, assigning the respective one of the nodes includes selecting a node that is connected to the respective port by a short path through the network relative to other nodes in the system.
Further preferably, configuring the network includes taking the switch being tested and the nodes assigned to the ports thereof off-line, while normal data transmission continues among other nodes and switches in the system.
Preferably, conveying the data includes transmitting data from at least one of the assigned nodes to at least one other of the assigned nodes through the corresponding ports at a substantially maximal transmission rate that the transmitting node can achieve. Further preferably, conveying the data includes conveying data through a selected pair of the ports of the switch being tested or, alternatively or additionally, conveying data through substantially all of the ports of the switch being tested.
Preferably, conveying the data includes conveying data packets, wherein observing the behavior of the switch includes comparing a number of data packets sent by at least one of the assigned nodes, to the number of packets received by one or more other assigned nodes and noting a discrepancy between the numbers. In a preferred embodiment, in normal operation of the system, the data packets are routed between any pair of the nodes over a plurality of different routes in alternation, and conveying the data includes routing substantially all of the packets conveyed from a first one of the assigned nodes to a second one of the assigned nodes over one assigned route through the switch being tested. Most preferably, the nodes are linked to the network by respective data link adapters, each of which routes data from the respective node through the network in accordance with a routing table stored in a memory thereof, and wherein routing substantially all of the packets includes downloading to the adapter of the first one of the assigned nodes a test routing table containing the assigned route.
There is additionally provided, in accordance with a preferred embodiment of the present invention, in a computer network system that includes a multiplicity of nodes interconnected by a network of switches, wherein the nodes are linked to the network by respective data link adapters, each of which in normal operation routes data packets from a corresponding one of the nodes through the network to a given destination node over a plurality of different routes in alternation, a method for testing of the system, including:
selecting at least one of the data link adapters;
programming the selected data link adapter so that substantially all of the data packets sent from the corresponding node to the given destination node are routed in repetition over a single route;
conveying data from the corresponding node to the destination node over the single route; and
observing operation of the system responsive to the conveyed data so as to identify a fault in the system.
Preferably, the selected data link adapter routes the data packets in accordance with a routing table stored in a memory, and programming the selected data link adapter includes downloading a test routing table to the memory, responsive to which the data packets are routed in repetition over the single route. In a preferred embodiment, the routing table normally contains multiple entries corresponding to the given destination node, each entry indicating one of the plurality of different routes, from among which entries the switch adapter in normal operation chooses the routes in alternation, and downloading the test routing table includes downloading a table including a plurality of identical entries corresponding to the given destination node.
There is moreover provided, in accordance with a preferred embodiment of the present invention, in a computer network system that includes a multiplicity of nodes interconnected by a network of switches, wherein the nodes are linked to the network by respective data link adapters, each of which routes data from the respective node through the network in accordance with a normal routing table stored in a memory thereof, a method for testing of the system, including:
selecting at least one of the data link adapters;
downloading to the selected data link adapter a respective test routing table, which differs substantially from the normal routing table thereof;
conveying data from the selected data link adapter through the network in accordance with the test routing table; and
observing operation of the system responsive to the conveyed data so as to identify a fault therein.
There is furthermore provided, in accordance with a preferred embodiment of the present invention, a manageable computer network system, including:
a network of switches, each switch having multiple ports; and
a multiplicity of nodes interconnected by the switches, wherein the nodes include a management node, which carries out a diagnostic test of a selected one of the switches by assigning a respective one of the nodes to each of a plurality of the ports of the selected switch, and by configuring the network so that the selected switch is substantially isolated from data flow from and to the remaining nodes in the network that are not assigned to the ports of the switch, and which issues a command to one or more of the nodes assigned to the ports to perform the test by conveying data at a controlled rate to another one or more of the assigned nodes through the respective ports, and receives information from the nodes in response to the test in order to detect a fault of the selected switch.
Preferably, the management node takes the selected switch and the nodes assigned to the ports thereof off-line, while normal data transmission continues among other nodes and switches in the system.
Further preferably, the network includes a packet-switched network.
There is also provided, in accordance with a preferred embodiment of the present invention, a manageable computer network system, including:
a multiplicity of nodes;
a network of switches, each switch having multiple ports; and
a multiplicity of data link adapters, each linking a respective one of the nodes to one of the ports of one of the switches, each of which adapters in normal operation routes data packets from the respective node through the network to a given destination node over a plurality of different routes in alternation,
wherein in a test mode of the system, at least one of the data link adapters is selected and programmed so that substantially all of the data packets sent from the corresponding node to the given destination node are routed in repetition over a single route, so that operation of the system can be observed while data are conveyed over the single route in order to identify a fault in the system.
Preferably, the selected data link adapter includes a memory and routes the data packets in accordance with a routing table stored in the memory, and a test routing table is downloaded to the selected adapter, responsive to which the data packets are routed in repetition over the single route. Further preferably, the routing table normally contains multiple entries corresponding to the given destination node, each entry indicating one of the plurality of different routes, and from among which entries the switch adapter in normal operation chooses the routes in alternation, and the test routing table includes a plurality of identical entries corresponding to the given destination node.
There is additionally provided, in accordance with a preferred embodiment of the present invention, a computer software product for testing a selected one of a plurality of switches, each switch having multiple ports, in a computer network system including a multiplicity of processor nodes coupled to the switches, one of which nodes is designated a primary node, the product including computer-readable code, which is read by the primary node, causing the primary node to assign a respective one of the nodes to each of a plurality of the ports of the selected switch, to configure the network so that the switch being tested is substantially isolated from data flow from and to the remaining nodes in the network that are not assigned to the ports of the switch, to cause data to be conveyed at a controlled rate between two of more of the assigned nodes through the respective ports of the switch, and to observe behavior of the switch while the data are being conveyed.
There is furthermore provided, in accordance with a preferred embodiment of the present invention, a computer software product for performing diagnostic testing in a computer network system that includes a multiplicity of nodes interconnected by a network of switches, wherein the nodes are linked to the network by respective data link adapters, each of which in normal operation routes data packets from a corresponding one of the nodes through the network to a given destination node over a plurality of different routes in alternation, the product including computer-readable code, which is read by a selected node among the multiplicity of nodes and causes the node to program a selected data link adapter so that substantially all of the data packets sent from the corresponding node to the given destination node are routed in repetition over a single route, whereby when data are transmitted from the corresponding node to the destination node over the single route, operation of the system responsive to the conveyed data is observed so as to identify a fault in the system.
The present invention will be more fully understood from the following detailed description of the preferred embodiments thereof, taken together with the drawings in which: